Generally, the viewer of a video stream has no control over the perspective used to record the video, whether it is a live transmission or a playback from a recording. The perspective is generally referred to as the viewpoint of the video.
In many cases a viewer may be interested to see the events displayed from a different viewpoint, for example when viewing a soccer game the viewer may wish to see the game-play from a different angle or to replay a segment of game play from various angles.
A number of methods have been implemented to provide viewers with different views of a video recording; some may be applied during live transmissions and some are applied during video playback. Some of the methods allow the viewer to select the viewpoint and some select the viewpoint for the viewer from a plurality of available views.
A first method provides a setup with multiple cameras, wherein the direction in which each camera is pointed can be controlled by a remote viewer and the viewer can select one of the cameras for viewing. A system such as this may be implemented, for example in a security surveillance system. However such a system can only be controlled by a single viewer and it is limited to the viewpoint of the selected camera.
A second method provides the use of multiple cameras that are positioned at different angles to record a live event. A director decides, which camera will be displayed to the viewers during the event, and a cameraman can change the viewing angle of a specific camera. This type of system is generally implemented in live television broadcasts. However such a system does not provide the viewer with any control over the viewpoint.
A third method provides the use of multiple cameras to record an event, and provides the viewer with all or part of the recorded video streams to select from to view. Such a system is commonly implemented in manufacturing DVD video records, wherein the viewer can select to view specific scenes from multiple viewpoints. However this method generally provides a limited number of viewpoints (e.g. 2-3), only for specific scenes of the entire film, and does not offer continuous transition between the available viewpoints.
A fourth method provides the use of one or more panoramic cameras that can provide images from multiple directions surrounding the panoramic camera, some in the direction of interest and some not. Such a system is commonly incorporated to provide a live view to viewers over the Internet. The viewer can select a camera and the direction of interest for viewing the video. The viewer is then provided with a video stream according to the selected parameters. This method is generally limited to a specific number of viewpoints of a specific subject and does not provide continuous transition between viewpoints around the subject.
A fifth method uses multiple cameras to record an event to a computer. The computer calculates the depth for each pixel of the recordings by comparing the images of the recordings from adjacent cameras. A remote viewer can request to view the event from any perspective. The computer performs complex graphical calculations to create the requested view based on the recordings and depth calculations from each camera on either side of the desired perspective, and transmits the results to the viewer. This method places a heavy calculation burden on the computer and is not adapted for scalability, since the computer is required to perform the complex calculations for each viewer. Alternatively, the computer could transmit all the data to each viewer to prepare its own viewpoint video, however this would require that the viewers all be equipped with strong computers.
U.S. Pat. No. 6,864,903 to Suzuki describes an Internet system for virtual tele-presence based on the fifth method described above, the disclosure of which is incorporated herein by reference. Suzuki also describes using an infrared camera with each regular camera to obtain the depth data required for the video produced by each regular camera without interfering with the recordings of the regular cameras.